Simulator, simulation method, and simulation program

ABSTRACT

One or more embodiments may provide a simulator for reproducing, in a virtual space, a motion of a machine that is controlled by a control program to manipulate an object. The simulator includes a processor configured with a simulation program to perform operations including: determining a condition of the object from a real space image; calculating an initial position of a virtual object in the virtual space corresponding to the determined condition; calculating a command value for moving a virtual machine in the virtual space, according to the control program and based on the position of the virtual object manipulated by the virtual machine in the virtual space, wherein the virtual machine corresponds to the machine; and creating display data for displaying one of: a motion of the virtual machine and a motion of the virtual object that are moved in accordance with the calculated command value.

This application is a continuation of application Ser. No. 14/385,172,filed on Feb. 10, 2015, which is based upon International ApplicationNo. PCT/JP2013/054157, filed on Feb. 20, 2013, which claims prioritybased on the Article 8 of Patent Cooperation Treaty from prior JapanesePatent Applications No. 2012-058273, filed on Mar. 15, 2012, the entirecontents of which is incorporated herein by reference.

TECHNICAL FIELD Background Art

There have been program creation apparatuses configured to create acontrol program to be used in an image processing device configured toextract measuring results from an image of a test object photographed bya camera (for example, Japanese Unexamined Patent Publication No.2009-123069, hereinafter, referred to as “Patent Document 1”). Theprogram creation apparatuses can realize off-line simulations bypreviously saving images photographed by real cameras as registeredimages and extracting measuring results from the registered images.

PRIOR ART DOCUMENTS Patent Documents

-   Patent Document 1: Japanese Unexamined Patent Publication No.    2009-123069

SUMMARY OF THE INVENTION Problems to be Solved by the Invention

However, the off-line simulation of Patent Document 1 cannot realize asimulation of synchronous operation of measuring results from visualsensors such as cameras and machine control on the same time axis.

The present invention is adapted to solve the above described problemand one of the objects of the present invention is to provide asimulator, a simulation method, and a simulation program which arecapable of realizing an integrated simulation that covers a visualsensor.

Means for Solving the Problem

In order to achieve the above described object, according to an aspectof the present invention, a simulator is an apparatus having a controlunit configured to perform a simulation of a control program executed ina controller that controls motion of a machine that manipulates anobject.

The control unit includes a first calculator configured to calculate acommand value for moving a virtual machine according to the controlprogram and based on model data of a virtual object wherein the virtualmachine corresponds to the machine which manipulates the object andwherein the virtual object is manipulated by the virtual machine in avirtual space and corresponds to the object manipulated by the machine;a second calculator configured to calculate motion of the virtualmachine which is moved in accordance with the command value calculatedby the first calculator; a third calculator configured to calculatemotion of the virtual object which is moved in accordance with themotion of the virtual machine calculated by the second calculator; and avirtual photographing part configured to generate a virtual space imagewhich is acquired in the case where the motion of the virtual machinecalculated by the second calculator or the motion of the virtual objectcalculated by the third calculator is virtually photographed. The firstcalculator is configured to calculate the command value further based onthe virtual space image generated by the virtual photographing part.

Preferably, the first calculator, the second calculator, the thirdcalculator, and the virtual photographing part respectively calculatethe command value, the motion of the virtual machine, and the motion ofthe virtual object and generate the virtual space image in accordancewith the same time axis.

Preferably, the control unit further includes a determination partconfigured to determine a condition of the object or the machine from areal space image photographed by a visual sensor in a real spacecorresponding to the virtual photographing part; and a fourth calculatorconfigured to calculate an initial condition of the object or themachine from the condition determined by the determination part. Thefirst calculator is configured to calculate the command value on theassumption that the initial condition calculated by the fourthcalculator is a condition at a starting time of the simulation.

Preferably, the simulator further includes a storage part and a display.The storage part includes an image storage part configured to associatethe real space image photographed by the visual sensor in the real spacecorresponding to the virtual photographing part with the virtual spaceimage corresponding to the real space image and to store the associatedreal space image and virtual space image beforehand. The control unitincludes a display controller configured to cause the display to displaythe real space image where the real space image is stored in the imagestorage part in association with the virtual space image generated bythe virtual photographing part.

According to another aspect of the present invention, a simulationmethod is a method performed in a simulator including a control unitconfigured to perform a simulation of a control program executed in acontroller that controls motion of a machine that manipulates an object.

In the simulation method, the control unit includes: a first step tocalculate a command value for moving a virtual machine according to thecontrol program and based on model data of a virtual object where thevirtual machine corresponds to the machine which manipulates the objectand the virtual object is manipulated by the virtual machine in avirtual space and corresponds to the object manipulated by the machine;a second step to calculate motion of the virtual machine which is movedin accordance with the command value calculated in the first step; athird step to calculate motion of the virtual object which is moved inaccordance with the motion of the virtual machine calculated in thesecond step; and a virtual photographing step to generate a virtualspace image which is acquired in the case where the motion of thevirtual machine calculated in the second step or the motion of thevirtual object calculated in the third step is virtually photographed.The first step includes a step to calculate the command value based onthe virtual space image generated in the virtual photographing step.

According to yet another aspect of the present invention, a simulationprogram is a program executed in a simulator including a control unitconfigured to perform a simulation of a control program executed in acontroller that controls motion of a machine that manipulates an object.

The simulation program causes the control unit to perform: a first stepof calculating a command value for moving a virtual machine according tothe control program and based on model data of a virtual object wherethe virtual machine corresponds to the machine which manipulates theobject and the virtual object is manipulated by the virtual machine in avirtual space and corresponds to the object manipulated by the machine;a second step of calculating motion of the virtual machine which ismoved in accordance with the command value calculated in the first step;a third step of calculating motion of the virtual object which is movedin accordance with the motion of the virtual machine calculated in thesecond step; and a virtual photographing step of generating a virtualspace image which is acquired in the case where the motion of thevirtual machine calculated in the second step or the motion of thevirtual object calculated in the third step is virtually photographed.The first step includes a step of calculating the command value based onthe virtual space image generated in the virtual photographing step.

Effect of the Invention

According to the present invention, an integrated simulation that coversa visual sensor can be realized. Further, a test in the case where avisual sensor is used in machine control can be carried out.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a diagram illustrating a configuration of a control systemaccording to an embodiment of the present invention.

FIG. 2 is a diagram illustrating a hardware configuration of a PCaccording to the embodiment of the present invention.

FIG. 3 is a diagram illustrating a functional block realized by a CPUexecuting a simulation program.

FIG. 4 is a first illustration showing a simulation state in theembodiment of the present invention.

FIG. 5 is a flow chart showing a control flow of the simulation in thefirst embodiment.

FIG. 6 is a diagram illustrating a 3D space of the simulation in thefirst embodiment.

FIG. 7 is a diagram for describing a control based on positions ofvirtual workpieces recognized by a virtual visual sensor in the firstembodiment.

FIG. 8 is a flow chart showing a control flow of the simulation in asecond embodiment.

FIG. 9 is a diagram illustrating a 3D space of the simulation in thesecond embodiment.

FIG. 10 is a diagram for describing a control based on positions ofvirtual workpieces recognized by the virtual visual sensor in the secondembodiment.

FIG. 11 is a flow chart showing a control flow of the simulation in athird embodiment.

FIG. 12 is a diagram illustrating a 3D space of the simulation in thethird embodiment.

MODE FOR CARRYING OUT THE INVENTION

Embodiments of the present invention will be described in detail belowwith reference to the drawings. The same and corresponding parts of thedrawings are denoted by the same reference symbols and the descriptionthereof will not be repeated.

First Embodiment

FIG. 1 is a diagram illustrating a configuration of a control systemaccording to an embodiment of the present invention. Referring to FIG.1, the control system according to the embodiment of the presentinvention includes a server 2, a network 4, a PC (Personal Computer) 6,a controller 14, and a control target device 16.

The server 2 is connected to the PC 6 via the network 4. The PC 6 iscommunicatively connected to the controller 14 which controls thecontrol target device 16.

The PC 6 is corresponding to the simulator in an embodiment of thepresent invention. A controller support program 8 including thesimulation program is installed in the PC 6 and a control program 10created by a user is stored in the PC 6. A CD-ROM (Compact Disc-ReadOnly Memory) 12 stores the controller support program 8. The controllersupport program 8 installed in the PC 6 has been installed from theCD-ROM 12.

The controller 14 controls operations of the control target device 16.In the embodiment of the present invention, a PLC (Programmable LogicController) is used as an example of the controller 14. That is, aso-called motion control function is provided in the PLC. The controller14 stores a control program 15 that defines details of control exertedon the control target device 16. The controller 14 executes the controlprogram 15 once in each control period. In this embodiment, the controlprogram 15 stored in the controller 14 is copied data of the controlprogram 10 stored in the PC 6 and is transmitted from the PC 6.

The control target device 16 includes a motor 18 such as a servo-motoror a stepper motor, and a motor driver 17 which drives a motor.

The motor 18 receives a driving current supplied from the motor driver17. The controller 14 which executes the control program 15 provides acommand value about a position for the motor driver 17 in each controlperiod, and the motor driver 17 supplies the driving current accordingto the command value to the motor 18. In the case where the motor 18 isa servo-motor, the motor 18 is provided with an encoder which detects anactual measurement value of rotational position of the motor 18. Theactual measurement value of rotational position of the motor is used bythe motor driver 17 in feedback control.

Incidentally, although the case where the simulation program isinstalled in the PC 6 via the CD-ROM 12 has been described above, thepresent invention is not limited to that and the simulation program maybe downloaded from the server 2 via the network 4 to the PC 6. The sameapplies to the control program.

FIG. 2 is a diagram illustrating a hardware configuration of the PC 6according to the embodiment of the present invention. Referring to FIG.2, the PC 6 according to the embodiment of the present inventionincludes a CPU 901 which is a processor, a ROM 902, a RAM 903, and anHDD 904 which are storage parts, a CD-ROM drive unit 908 which is a datareader, a communication IF 909 which is a communication part, a monitor907 which is a display, and a keyboard 905 and a mouse 906 which areinput parts. Meanwhile, these components are interconnected via aninternal bus 910.

The HDD 904, which is typically a non-volatile magnetic memory, storesthe simulation program read out by the CD-ROM drive unit 908 from theCD-ROM 12. The HDD 904 also stores the control program 15.

The CPU 901 loads the controller support program 8 according to theembodiment stored in the HDD 904 into the RAM 903 and executes thecontroller support program 8.

The RAM 903 is a volatile memory and used as a work memory. The ROM 902generally stores programs including an operating system (OS).

The communication IFs 909, which typically support general-purposecommunication protocols such as Ethernet (Registered Trademark) or USB(Universal Serial Bus), provide data communication between the PC 6 andthe server 2 via the network 4 while providing data communicationbetween the PC 6 and the controller 14.

The monitor 907 is constituted of a liquid crystal display device, a CRT(Cathode Ray Tube), a plasma display device, and the like and displaysresults processed by the PC 6 and the like. The keyboard 905 receiveskey entries performed by a user, whereas the mouse 906 receives pointingoperations performed by the user.

FIG. 3 is a diagram illustrating a functional block realized by the CPU901 executing the controller support program 8. Referring to FIG. 3, auser interface part 802, a display data creation part 804, a simulationpart 806, a control program storage part 808, a control program editor810, and a controller interface part 812 are illustrated.

The user interface part 802 is a component configured to create contentsof a window to be displayed on the monitor 907 of the PC 6 and receive auser operation performed on the keyboard 905 and the mouse 906.

The control program editor 810 allows the user to input and edit thecontrol program. The control program editor 810 is also responsible forcompilation in the case where the control program requires to becompiled for execution. The created control program is sent to thecontroller 14 via the controller interface part 812. Further, thecreated control program is stored in the control program storage part808 which is a predetermined area of the HDD 904. The control programeditor 810 can also read out the control program 15 stored in thecontroller 14 via the controller interface part 812 to edit the controlprogram 15.

The simulation part 806 is a simulator of the controller 14. Thesimulation part 806 simulates operations of the controller 14 executingthe control program 15 according to the control program 10 stored in thecontrol program storage part 808 and calculates the command value aboutthe position to be output in each control period from the controller 14.

Further, the simulation part 806 can perform simulations such as asimulation of a situation in which arrival of a signal from outsideaffects operations of the control program and a simulation of asituation in which execution of the control program 15 changes aninternal condition of the controller 14 including contents stored in amemory of the controller 14 and consequently the change affectsoperations of the control program 15.

Still further, the simulation part 806 receives an instruction from theuser about execution of a simulation via the user interface part 802. Inother words, the user interface part 802 also functions as a componentfor receiving an instruction from the user destined to the simulationpart 806.

The display data creation part 804 creates display data for displaying achronological change in execution result data created by the simulationpart 806. The display data creation part 804 causes the created displaydata to be displayed on the monitor 907 of the PC 6 in the forms ofgraph and text or in the form of 3D representation by sending thecreated display data to the user interface part 802.

FIG. 4 is a first illustration showing a simulation state in theembodiment of the present invention. Referring to FIG. 4, in theembodiment, a virtual conveyor 520 corresponding to a conveyor in a realspace, a virtual visual sensor 510 corresponding to a visual sensor inthe real space, and a virtual workpiece 540 corresponding to a workpiecein the real space are arranged in a 3D space 500 which is a virtualspace corresponding to the real space.

The simulation part 806 illustrated in FIG. 3 executes a 3D simulator, avisual sensor simulator, and a machine control simulator.

The 3D simulator causes objects (in this embodiment, the virtual visualsensor 510, the virtual conveyor 520, and the virtual workpiece 540) tobe displayed in the 3D space 500 based on the result acquired by datatrace.

The visual sensor simulator performs image processing on virtual imagedata 570 of the object acquired in the 3D space 500 (in this embodiment,the virtual workpiece 540) and reflects a determination result of theimage processing on control of a virtual machine which is a virtualmachine in the 3D space 500 (in this embodiment, the virtual conveyor520).

The machine control simulator controls the virtual machine which ispresent in the 3D space 500. Specifically, the machine control simulatorcalculates the command value for the control of the virtual machine andcalculates motion of the virtual machine corresponding to the commandvalue.

Comparing the virtual image data 570 with a real image data 590 realizessynchronization of the 3D simulator, the visual sensor simulator, andthe machine control simulator.

Specifically, as one of methods of comparing the virtual image data 570with the real image data 590, there is a method of combining imageprocessing that is for extracting a contour of a workpiece (object) fromthe virtual image data 570 and a contour of a workpiece (object) fromthe real image data 590 with processes of comparing the extracted twosets of contours with each other and determining whether the contoursfit with each other. Further, there is a method of combining a processof converting the real image data 590 into a gray-scale image byremoving color data from the real image data 590 with processes ofcalculating approximate values of the gray-scale image after theconversion and the virtual image data 570 and determining whether theapproximate values agree with each other.

Alternatively, as one of methods of synchronizing the 3D simulator, thevisual sensor simulator, and the machine control simulator by using aresult of comparison of the virtual image data 570 with the real imagedata 590, there is a method of causing the 3D simulator, the visualsensor simulator, and the machine control simulator to function inconjunction with each other such that (1) the machine control simulatorissues a photographing instruction, (2) the visual sensor simulatorresponds to the photographing instruction by issuing a request toacquire the virtual image data 570 in the 3D space 500, (3) the 3Dsimulator responds to the acquisition request by sending the virtualimage data 570 of a predetermined photographing position, (4) the visualsensor simulator recognizes a workpiece position by using the result ofcomparison of the virtual image data 570 with the real image data 590based on the received virtual image data 570 and sends the recognizedworkpiece position, and (5) the machine control simulator controls thevirtual machine based on the received workpiece position.

FIG. 5 is a flow chart showing a control flow of the simulation in thefirst embodiment. Referring to FIG. 5, in step S101, the simulation part806 virtually arranges a sample object for calibration at apredetermined calibration position in the 3D space 500 by executing the3D simulator.

In step S201, by executing visual sensor simulator, the simulation part806 virtually photographs the sample object arranged at thepredetermined calibration position in the 3D space 500 with the 3Dsimulator and performs calibration. In the calibration, whether aphotographed image the same as a previously saved image of apredetermined sample object can be obtained or not is confirmed and inthe case where a photographed image the same as the previously savedimage cannot be obtained, the range of image to be obtained and the likeare adjusted so that a photographed image the same as the previouslysaved image can be obtained.

Then in step S311, the simulation part 806 sets an initial position of aworkpiece by executing the machine control simulator.

FIG. 6 is a diagram illustrating the 3D space 500 of the simulation inthe first embodiment. Referring to FIG. 6, in the embodiment, virtualworkpieces 540A-540D are set on the virtual conveyor 520, and as aresult that a belt of the virtual conveyor 520 is driven, the virtualworkpieces 540A-540D move from the left to the right-hand side in FIG. 6to be handled by virtual robots 530A and 530B in the 3D space 500corresponding to robots in a real space. Further, the virtual workpieces540A-540D are photographed by the virtual visual sensor 510.

The initial positions of the virtual workpieces 540A-540D are set on thevirtual conveyor 520 on the left of the virtual workpieces 540A-540Dwhich are illustrated in solid lines in the drawing.

Returning to FIG. 5, next in step S312, by executing the machine controlsimulator, the simulation part 806 starts controlling the virtualmachines (in this embodiment, the virtual conveyor 520, the virtualrobots 530A and 530B) by starting execution of the control program 15.

In step S313, the simulation part 806 performs sequence control based onthe positions of the workpieces. In step S314, the simulation part 806performs motion control based on the positions of the workpieces.

Then in step S315, the simulation part 806 calculates conditions of themachines and the workpieces resulting from the motion control, and instep S316, the simulation part 806 sends the calculated conditions ofthe machines and the workpieces to the 3D simulator.

Next in step S112, by executing the 3D simulator, the simulation part806 receives the conditions of the machines and the workpieces sent fromthe machine control simulator, and in step S113, execution result datawhich is necessary to cause the machines and the workpieces in thereceived conditions to be displayed in the 3D space 500 of the monitor907 is sent to the display data creation part 804.

Referring to FIG. 6 again, as a result of the sequence control and themotion control performed, the virtual workpieces 540A-540D are carriedrightward by the virtual conveyor 520 and lifted up and set at otherplaces by the virtual robots 530A and 530B. That situation is displayedon the monitor 907 as illustrated in FIG. 6.

Returning to FIG. 5, next, the simulation part 806 determines whetherthe present point in time is photographing timing or not by executingthe machine control simulator. In FIG. 6, as a result of determinationof whether a current position of an encoder shaft of the virtualconveyor 520 has reached a specified position or not, it is determinedwhether the present point in time is photographing timing or not.Alternatively, a virtual photoelectric tube sensor may be separatelyprovided so that the photographing timing is decided as a moment whenthe virtual workpiece blocks an optical axis.

In the case where it is determined that the present point in time is notthe photographing timing (in the case where it is determined NO in stepS317), the simulation part 806 returns the process to be performed tothe process of step S313.

On the other hand, in the case where it is determined that the presentpoint in time is the photographing timing (in the case where it isdetermined YES in step S317), in step S318, the simulation part 806sends a photographing instruction to the visual sensor simulator.

In step S212, by executing the visual sensor simulator, the simulationpart 806 determines whether the photographing instruction sent from themachine control simulator is received or not. In the case where it isdetermined that the photographing instruction is not received (in thecase where it is determined NO in step S212), the simulation part 806repeats the process of step S212.

On the other hand, in the case where it is determined that thephotographing instruction is received (in the case where it isdetermined YES in step S212), in step S213, by executing the visualsensor simulator, the simulation part 806 sends an image acquisitionrequest to acquire the virtual image data 570 in the 3D space 500 to the3D simulator.

In step S114, by executing the 3D simulator, the simulation part 806determines whether the image acquisition request sent from the visualsensor simulator is received or not. In the case where it is determinedthat the image acquisition request is not received (in the case where itis determined NO in step S114), the simulation part 806 returns theprocess to be performed to the process of step S112.

On the other hand, in the case where it is determined that the imageacquisition request is received (in the case where it is determined YESin step S114), in step S115, the simulation part 806 sends the virtualimage data 570 of a predetermined photographing position to the visualsensor simulator. The predetermined photographing position refers to aposition in the 3D space 500 corresponding to a position in the realspace at which the visual sensor in the real space corresponding to thevirtual visual sensor 510 is aimed. After step S115, the simulation part806 returns the process to be performed to the process of step S112.

In step S214, by executing the visual sensor simulator, the simulationpart 806 receives the virtual image data 570 of a predeterminedphotographing position sent from the 3D simulator.

Next in step S215, by executing the visual sensor simulator, thesimulation part 806 recognizes the workpiece positions from the virtualimage data 570, and in step S216, the simulation part 806 sends therecognized workpiece positions to the machine control simulator andreturns the process to be performed to the process of step S212.

In step S319, by executing the machine control simulator, the simulationpart 806 receives the workpiece positions sent from the visual sensorsimulator and returns the process to be performed to the process of stepS313. Based on the received workpiece positions, the above describedprocesses of step S313 and step S314 are performed.

Referring to FIG. 6 again, the virtual robots 530A and 530B pick up thevirtual workpieces 540A-540D based on the positions of the virtualworkpieces 540A-540D recognized from the virtual image data 570 from thevirtual visual sensor 510 and set the virtual workpieces 540A-540D atother places, for example.

FIG. 7 is a diagram for describing a control based on the positions ofthe virtual workpieces 540A-540D recognized by the virtual visual sensor510 in the first embodiment. Referring to FIG. 7, as a result of stepS115 of the 3D simulator in FIG. 5 executed, the virtual image data 570that is assumed to be photographed by the virtual visual sensor 510 iscreated. In the virtual image data 570, a situation in which the virtualworkpieces 540A-540D are set on the virtual conveyor 520 isphotographed.

Then, as a result of step S215 of the visual sensor simulator in FIG. 5executed, the virtual image data 570 is recognized and the positions ofthe virtual workpieces 540A-540D are determined.

Next, as a result of step S313 and step S314 of the machine controlsimulator in FIG. 5 executed, the command values for the virtual robots530A and 530B are calculated and the motion control is performed.

Then, as a result of step S113 of the 3D simulator in FIG. 5 executed, asituation in which the virtual workpieces 540A-540D are being handled bythe virtual robots 530A and 530B is displayed in the 3D space 500 of themonitor 907.

Second Embodiment

FIG. 8 is a flow chart showing a control flow of the simulation in thesecond embodiment. Referring to FIG. 8, in the second embodiment, afterstep S101 described in FIG. 5 of the first embodiment, in step S102, byexecuting the 3D simulator, the simulation part 806 reads the real imagedata 590 of a predetermined photographing position photographed by thevisual sensor in the real space corresponding to the virtual visualsensor 510.

Next in step S103, the simulation part 806 determines the workpieceposition in a predetermined photographing position (in this embodiment,a position at which the virtual visual sensor 510 is aimed in the 3Dspace 500, and a position at which the visual sensor is aimed in thereal space) from the read real image data 590.

FIG. 9 is a diagram illustrating the 3D space 500 of the simulation inthe second embodiment. Referring to FIG. 9, the predeterminedphotographing position of the virtual visual sensor 510 (in the drawing,the part surrounded by a dashed parallelogram) is determined from thereal image data 590 corresponding to the virtual image data 570.

Returning to FIG. 8, in step S104, the simulation part 806 calculatesthe initial position of the workpiece from the position of the workpiecedetermined in step S103. The initial position of the workpiece may be aposition determined from the real image data 590 or a position previousto the position determined from the real image data 590.

Then, in step S105, the simulation part 806 sends the initial positionof the workpiece calculated in step S104 to the machine controlsimulator. Since the processes of step S112 onward after step S105 arethe same as the processes in FIG. 5 of the first embodiment, redundantdescription of the processes will not be repeated.

Next, by executing the machine control simulator, in step S301, thesimulation part 806 receives the initial position of the workpiece sentfrom the 3D simulator, and in step S311A, the simulation part 806 setsthe received initial position of the workpiece as an initial position ofthe workpiece to be simulated. Since the processes of step S312 onwardafter step S311A are the same as the processes in FIG. 5 of the firstembodiment, redundant description of the processes will not be repeated.

Consequently, as a result of execution of the simulation from theposition based on the real image data 590, a situation in the real spacecan be reproduced. Further, in the case where the real image data 590read in step S102 is the real image data 590 on the occurrence of atrouble, the situation on the occurrence of the trouble can bereproduced. For example, in the case where the real image data 590 of afailure in lifting up of a workpiece is read in, a situation of thefailure in lifting up of the workpiece can be reproduced.

FIG. 10 is a diagram for describing a control based on the positions ofthe virtual workpieces 540A-540D recognized by the virtual visual sensor510 in the second embodiment. Referring to FIG. 10, as a result of stepS103 and step S104 of the 3D simulator in FIG. 8 executed, contours ofthe workpieces are extracted from the real image data 590 so thatworkpiece models including positions and a shape of the workpieces aredetermined and generated. Since the flow onward is the same as the flowin the first embodiment described with reference to FIG. 7, redundantdescription of the flow will not be repeated.

Third Embodiment

FIG. 11 is a flow chart showing a control flow of the simulation in thethird embodiment. Referring to FIG. 11, in the third embodiment, afterstep S113 described in FIG. 5 of the first embodiment, in step S121, byexecuting the 3D simulator, the simulation part 806 reads the real imagedata 590, which is corresponding to image data approximating to thevirtual image data 570 of a predetermined photographing position in the3D space 500 corresponding to a position at which the visual sensor inthe real space is aimed, out from the HDD 904. Incidentally, the virtualimage data 570 in the 3D space 500 and the corresponding real image data590 in the real space are associated with each other and storedbeforehand in the HDD 904.

Next in step S122, the simulation part 806 sends the real image data 590which is necessary to cause the real image represented by the real imagedata 590 read out in step S121 to be displayed on the monitor 907 to thedisplay data creation part 804. Since the loop processes from step S112to step S115 including step S122 are executed in a very short period(for example, by a few milliseconds to by tens of milliseconds), thereal image is displayed as a moving image on the monitor 907. Since theprocesses of step S114 onward after step S122 are the same as theprocesses in FIG. 5 of the first embodiment, redundant description ofthe processes will not be repeated.

FIG. 12 is a diagram illustrating the 3D space 500 of the simulation inthe third embodiment. Referring to FIG. 12, as illustrated in thedrawing, the real image data 590A-590C which is associated with imagedata approximating to the virtual image data 570C-570C and storedbeforehand replaces the virtual image data 570A-570C in the 3D space 500and is displayed as a moving image of real images.

Conclusion

(1) As described above, the PC 6 which is the simulator according to theaforementioned embodiments is an apparatus provided with a control unit(for example, the CPU 901) that performs a simulation of a controlprogram (for example, the control program 10, the control program 15)executed in a controller (for example, the controller 14) that controlsmotion of a machine (for example, the virtual conveyor 520, the virtualrobots 530, 530A, 530B) that manipulates an object (for example, thevirtual workpieces 540, 540A-540D).

The control unit includes a first calculator, a second calculator, athird calculator, and a virtual photographing part. The first calculatoris a component that calculates a command value for moving a virtualmachine (for example, the virtual conveyor 520, the virtual robots 530,530A, and 530B in the 3D space 500) according to the control program andbased on model data of a virtual object (for example, the workpiece inthe 3D space) where the virtual machine corresponds to the machine whichmanipulates the object and the virtual object is manipulated by thevirtual machine in a virtual space (for example, the 3D space 500) andcorresponds to the object manipulated by the machine (for example, thecomponent formed in the CPU 901 as a result of execution of step S313and step S314 in FIG. 5).

The second calculator is a component that calculates motion of thevirtual machine which is moved in accordance with the command valuecalculated by the first calculator (for example, the component formed inthe CPU 901 as a result of execution of step S315 in FIG. 5).

The third calculator is a component that calculates motion of thevirtual object which is moved in accordance with the motion of thevirtual machine calculated by the second calculator (for example, thecomponent formed in the CPU 901 as a result of execution of step S315 inFIG. 5).

The virtual photographing part is a component that generates an image ofa predetermined photographing position which is acquired in the casewhere the motion of the virtual machine calculated by the secondcalculator or the motion of the virtual object calculated by the thirdcalculator is virtually photographed (for example, the component formedin the CPU 901 as a result of execution of the virtual visual sensor 510and step S115 in FIG. 5).

The first calculator calculates the command value further based on thevirtual space image generated by the virtual photographing part (forexample, in step S215, the position of the workpiece is recognized fromthe virtual image data 570 of the predetermined photographing position,and in step S313 and step S314, the machine is controlled and thecommand value for the machine is calculated based on the position of theworkpiece).

As described above, the command value for moving the virtual machine iscalculated according to the control program and based on the model dataof the virtual object where the virtual machine corresponds to a machinewhich manipulates an object and the virtual object is manipulated by thevirtual machine in a virtual space and corresponds to the objectmanipulated by the machine; motion of the virtual machine which is movedin accordance with the calculated command value is calculated; motion ofthe virtual object which is moved in accordance with the calculatedmotion of the virtual machine is calculated; a virtual space image isgenerated where the virtual space image is assumed to be acquired in thecase where the calculated motion of the virtual machine or thecalculated motion of the virtual object is virtually photographed; andthe command value is calculated further based on the generated virtualspace image.

As a result, the virtual machine in the virtual space corresponding tothe machine in the real space is controlled on the basis of the virtualspace image which is generated by the virtual photographing part andcorresponds to the real space image photographed by the visual sensor inthe real space. An integrated simulation of a machine system including avisual sensor in a real space corresponding to a virtual photographingpart can be realized. Further, a test in the case where a visual sensoris used in machine control can be carried out.

(2) Further, the first calculator, the second calculator, the thirdcalculator, and the virtual photographing part respectively calculatethe command value, the motion of the virtual machine, and the motion ofthe virtual object and generate the virtual space image in accordancewith the same time axis (for example, the machine control simulator, the3D simulator, and the visual sensor simulator in FIG. 5 execute therespective loop processes by exchanging data with each other, therefore,the machine control simulator, the 3D simulator, and the visual sensorsimulator are synchronized with each other based on the timing of theexchange and operate on the common time axis). As a result, asynchronized integrated simulation can be realized.

(3) Further, the control unit also includes a determination part and afourth calculator. The determination part is a component that determinesa condition of the object or the machine from the real space imagephotographed by the visual sensor in the real space corresponding to thevirtual photographing part (for example, the component formed in the CPU901 as a result of execution of step S103 in FIG. 8).

The fourth calculator is a component that calculates an initialcondition of the object or the machine from the condition determined bythe determination part (for example, the component formed in the CPU 901as a result of execution of step S104 in FIG. 8).

The first calculator calculates the command value on the assumption thatthe initial condition calculated by the fourth calculator is a conditionat a starting time of the simulation (for example, in step S311A in FIG.8, an initial value is set, and in step S313 and step S314 in FIG. 5,the machine is controlled and the command value for the machine iscalculated).

As a result, the situation in the real space can be reproduced. In caseof an occurrence of a trouble, the situation on the occurrence of thetrouble can be reproduced.

(4) Further, the simulator also includes a storage part (for example,the RAM 903, the HDD 904) and a display (for example, the monitor 907).The storage part includes an image storage part that associates the realspace image photographed by the visual sensor in the real spacecorresponding to the virtual photographing part with the virtual spaceimage corresponding to the real space image and stores the associatedreal space image and virtual space image beforehand (for example, astorage area in which the virtual image data 570 in the 3D space 500 andthe corresponding real image data 590 in the real space are associatedwith each other and stored beforehand).

The control unit further includes a display controller. The displaycontroller is a component that causes the display to display the realspace image where the real space image is stored in the image storagepart in association with the virtual space image generated by thevirtual photographing part (for example, the component formed in the CPU901 as a result of execution of step S121 and S122 in FIG. 11).

As a result, motion of an object or a machine in the virtual space canbe displayed like an object or a machine in the real space.

Modification

(1) As a simulation of machine control by using the real image data 590photographed by the visual sensor in the real space, the aforementionedembodiments are adapted to perform the control of a virtual machine byusing the virtual image data 570 from the virtual visual sensor 510 inthe 3D space 500 which is a virtual space. In that case, the virtualvisual sensor 510 is configured to photograph the virtual workpieces540, 540A-540D which are virtual objects so that the virtual conveyor520 and the virtual robots 530A and 530B as virtual machines arecontrolled based on the virtual image data 570 of the photographedvirtual objects.

However, the present invention is not limited to that and the virtualvisual sensor 510 is configured to photograph the virtual machines suchas the virtual conveyor 520 or the virtual robots 530A and 530B so thatthe virtual machines are controlled based on the virtual image data 570of the photographed virtual machines.

Alternatively, the virtual visual sensor 510 may be configured tophotograph both of the virtual machines and the virtual objects so thatthe virtual machines are controlled based on the virtual image data 570of the photographed virtual machines and virtual objects.

(2) The aforementioned embodiments describe the case where the simulatorexecuted by the simulation part 806 is divided into three; the 3Dsimulator, the visual sensor simulator, and the machine controlsimulator.

However, the present invention is not limited to that and any two of thethree simulators may be integrated into one simulator or the threesimulators may be integrated into one simulator. Since that integrationof the simulators saves the simulators the exchange of data with eachother, the simulator can efficiently perform the simulations.

(3) The aforementioned embodiments describe the case where the virtualvisual sensor 510 corresponding to the visual sensor which copes withthe real image data 590 is used in the simulation. However, the presentinvention is not limited to that and any other sensor may be used ifonly the sensor can recognize the condition of the object or themachine: for example, an ultrasonic sensor, an optical sensor, aninfrared sensor, a temperature sensor, or a displacement sensor may beused instead of limiting the sensor to a visual sensor.

(4) The aforementioned embodiments are described as an invention of asimulator. However, the present invention is not limited to that and maybe understood as an invention of a simulation method performed in asimulator or as an invention of a simulation program executed in asimulator.

(5) The aforementioned third embodiment is adapted to cause the realimage data 590, which is corresponding to image data approximating tothe virtual image data 570 of a predetermined photographing position inthe 3D space 500 corresponding to a position at which the visual sensorin the real space is aimed, to be read out from the HDD 904 in step S121in FIG. 11 on the assumption that the virtual image data 570 in the 3Dspace 500 and the real image data 590 in the real space corresponding tothe virtual image data 570 are associated with each other and storedbeforehand.

However, the present invention is not limited to that and the real imagedata 590 photographed by the visual sensor in the real space may besaved in the HDD 904 as it is and the HDD 904 may be searched for thereal image data 590 which is to be compared with the virtual image data570 photographed by the virtual visual sensor 510 in real time when thesimulation is performed. As the comparing method, the methods describedwith reference to FIG. 4 in the first embodiment may be used.

The embodiments disclosed herein are illustrative in all aspects andshould not be construed as restrictive. The scope of the presentinvention is defined not by the above description but by the appendedclaims and all modifications within the equivalent meaning and scope ofthe appended claims are intended to be included in the invention.

DESCRIPTION OF SYMBOLS

-   -   2 server    -   4 network    -   8 controller support program    -   10, 15 control program    -   12 CD-ROM    -   14 controller    -   16 control target device    -   17 motor driver    -   18 motor    -   500 3D space    -   510 virtual visual sensor    -   520 virtual conveyor    -   530, 530A, 530B virtual robot    -   540, 540A-540D virtual workpiece    -   570, 570A-570C virtual image data    -   590, 590A-590C virtual image data    -   802 user interface part    -   804 display data creation part    -   806 simulation part    -   808 control program storage part    -   810 control program editor    -   812 controller interface part    -   901 CPU    -   902 ROM    -   903 RAM    -   904 HDD    -   905 keyboard    -   906 mouse    -   907 monitor    -   908 CD-ROM drive unit    -   909 communication IF    -   910 internal bus

The invention claimed is:
 1. A simulator for reproducing, in a virtualspace, a motion of a machine that is controlled by a control program tomanipulate an object, wherein the simulator comprises a processorconfigured with a simulation program that causes the processor toperform operations comprising: determining a condition of the objectfrom a real space image; calculating an initial position of a virtualobject in the virtual space corresponding to the determined condition;calculating a command value for moving a virtual machine in the virtualspace, according to the control program that controls the motion of themachine and based on the position of the virtual object manipulated bythe virtual machine in the virtual space, wherein the virtual machinecorresponds to the machine; and creating display data for displaying oneof: a motion of the virtual machine and a motion of the virtual objectthat are moved in accordance with the calculated command value.
 2. Thesimulator according to claim 1, wherein calculating the initial positionof the virtual object comprises calculating the initial position of thevirtual object by associating the real space image with a photographingposition of a virtual photographing part corresponding to a visualsensor that photographs the object.
 3. The simulator according to claim1, wherein creating the display data comprises creating the display datafor displaying the virtual object at the initial position.
 4. Thesimulator according to claim 1, wherein calculating the command valuecomprises calculating the command value, the motion of the virtualmachine, and the motion of the virtual object in accordance with thesame time axis, and creating the display data comprises creating thedisplay data for displaying the motion of the virtual machine and themotion of the virtual object.
 5. The simulator according to claim 1,wherein the processor is configured with the simulation program toperform operations further comprising: calculating a motion of thevirtual machine in accordance with the calculated command value; andcalculating a motion of the virtual object that is moved by thecalculated motion of the virtual machine, wherein creating the displaydata comprises creating the display data for displaying one of: thecalculated motion of the virtual machine and the calculated motion ofthe virtual object.
 6. The simulator according to claim 1, furthercomprising a display, wherein the processor is configured with thesimulation program to perform operations further comprising displayingon the display an image based on the created display data.
 7. Asimulation method performed in a simulator comprising a processorconfigured with a simulation program that causes the processor toreproduce, in a virtual space, a motion of a machine that is controlledby a control program to manipulate an object, wherein the methodcomprises: determining, by the processor, a condition of the object froma real space image; calculating, by the processor, an initial positionof a virtual object in the virtual space corresponding to the determinedcondition; calculating, by the processor, a command value for moving avirtual machine in the virtual space, according to the control programthat controls the motion of the machine and based on the position of thevirtual object manipulated by the virtual machine in the virtual space,wherein the virtual machine corresponds to the machine; and creating, bythe processor, display data for displaying one of: a motion of thevirtual machine and a motion of the virtual object that are moved inaccordance with the calculated command value.
 8. The method according toclaim 7, wherein calculating the initial position of the virtual objectcomprises calculating the initial position of the virtual object byassociating the real space image with a photographing position of avirtual photographing part corresponding to a visual sensor thatphotographs the object.
 9. The method according to claim 7, whereincreating the display data comprises creating the display data fordisplaying the virtual object at the initial position.
 10. The methodaccording to claim 7, wherein calculating the command value comprisescalculating the command value, the motion of the virtual machine, andthe motion of the virtual object in accordance with the same time axis,and creating the display data comprises creating the display data fordisplaying the motion of the virtual machine and the motion of thevirtual object.
 11. The method according to claim 7, further comprising:calculating, by the processor, a motion of the virtual machine inaccordance with the calculated command value; and calculating, by theprocessor, a motion of the virtual object that is moved by thecalculated motion of the virtual machine, wherein creating the displaydata comprises creating the display data for displaying one of: thecalculated motion of the virtual machine and the calculated motion ofthe virtual object.
 12. The method according to claim 7, furthercomprising displaying, by a display, an image based on the createddisplay data.
 13. A non-transitory computer-readable medium having asimulation program stored thereon, the simulation program causing asimulator comprising a processor to reproduce, in a virtual space, amotion of a machine controlled by a control program to manipulate anobject, the simulation program, when executed by the processor, causingthe processor to perform operations comprising: determining a conditionof the object from a real space image; calculating an initial positionof a virtual object in the virtual space corresponding to the determinedcondition; calculating a command value for moving a virtual machine inthe virtual space, according to the control program that controls themotion of the machine and based on the position of the virtual objectmanipulated by the virtual machine in the virtual space, wherein thevirtual machine corresponds to the machine; and creating display datafor displaying one of: a motion of the virtual machine and a motion ofthe virtual object that are moved in accordance with the calculatedcommand value.